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(54) Image-based touchscreen 

(57) In a touch activated computer system, a display 
terminal displays an image on a screen of the terminal. 
The image includes one or more visual option buttons 
arranged in a predetermined pattern. Each button cor- 
responds to a specific application program which can be 
executed by the system. A camera has a lens oriented 
to capture frames representing a scene in front of the 
screen. Each frame includes a plurality of regularized 
pixel values. The system includes an image registration 
subsystem for comparing the pixel values of successive 
frames. A particular application is selected for execution 
when the pixel values corresponding to that portion of 
the scene directly in front of a particular button differ in 
successive frames more than a predetermined thresh- 
old value. 
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Description 

FIELD OF THE INVENTION 

This invention relates to computer systems, and 
more particularly to touch activated display screens. 

BACKGROUND OF THE INVENTION 


In certain situations, it is advantageous to control 
the operation of computer systems using "touchscreen" 
technologies. For example, touch activated information 
systems in public places do not require a keyboard or 
mouse, or other similar hard-to-use and easy-to-dam- 
age control devices. 

In touch activated computer applications, the sys- 
tem displays images on a screen of a display terminal. 
The images can include visual option icons or "buttons, 
with appropriate legends. The user can activate proc- 
esses corresponding to the visual buttons by pointing 
at, or "touching" the screen where the buttons are dis- 
played. 

Previous touch activated systems have generally 
sensed the presence of a f inger, or pointing device on or 
near the display screen using capacitance, resistive, 
acoustic, mechanical, or optical sensors mounted 
directly on the screen, or around the periphery of the 
screen, for example, in the bezel of the display terminal. 

These sensing techniques generally require a 
physical modification of the display terminal. For exam- 
ple in capacitive and resistive systems, transparent lay- 
ers of conductive and dielectric films are placed directly 
on the screen. With systems that are acoustically or 
optically activated, the transmitters and receivers are 
generally mounted in the bezel or housing surrounding 

the screen. . 

The problem with the prior art touch technologies is 
that the display terminal must be physically modified at 
an increased cost to receive the sensors. Specialized 
sensor interfaces and software must also be generated. 
It is desirable to provide a touch activated system with- 
out the need to modify the display terminal. 

SUMMARY OF THE INVENTION 


While someone is "pointing" at the screen, the pixel 
values of a reference frame are compared with the pix- 
els of a subsequent selection frame. The frames can be 
compared using image registration techniques. 
5 A specific computer application is selected for exe- 
cution if the difference between the pixel values of a pre- 
determined portion of the reference frame, and a 
corresponding portion of the selection frame exceeds a 
predetermined threshold value. 
10 The invention, in its broad form, resides in a touch 
activated computer system and method, as recited in 
claims 1 and 9 respectively. 

Preferably, as described hereinafter, the camera is 
mounted above the display terminal with the lens point- 
15 ing in a downward direction. The optical axis of the lens 
is in from of the screen. This allows for the detection of 
pixel motion in vertically oriented stripes or zones. If the 
camera is mounted along side the terminal pointing 
sideways, then the detection stripes or zones can be 
20 arranged horizontally. 

Advantageously, a mirror is mounted in the seme 
viewed by the camera. The mirror is mounted at such an 
angle that the frames captured by the camera include a 
different view of the seen as reflected in the mirror. For 
25 example, if the mirror is mounted at an angle of about 
45° with respect to the optical axis, then the view in the 
mirror is substantially perpendicular to the directly 
viewed scene. Pixels corresponding to the different vi ew 
can be separately compared to allow application selec- 
30 tion using a two-dimensional arrangement of the visual 
display buttons. 


The invention enables the activation of computer 
applications by pointing at or touching specific portions 
of a screen of a display terminal. A computer program 
displays an image on the screen. The image includes 
one or more visual option buttons, each button corre- so 
sponding to a specific application program to be acti- 
vated. 

During operation of the system, a camera captures 
digitally encoded frames of a scene from of the screen. 
The frames can include calibration frames, and seiec- 55 
tion frames. Each frame includes a plurality of digitally 
encoded pixels arranged in a regularized pattern. The 
pixels encode light intensity values. 


BRIEF DESCRIPTION OF THE DRAWINGS 

35 A more detailed understanding of the invention can 
be had from the following description of preferred 
embodiments, given by way of example and to be read 
in conjunction with the accompanying drawing, wherein: 

40 ♦ Figure 1 is a block schematic of an image-based 
touch activated computer system according to a 
preferred embodiment the invention; 

♦ Figure 2 is an image shown on a screen of the sys- 
45 tern of Figure 1 ; 

+ Figures 3-5 represent frames grabbed by a camera 
of the system of Figure 1; 

♦ Figure 6 is a schematic diagram of a touch acti- 
vated system using a mirror; and 

♦ Figure 7 is a frame grabbed by the system of Figure 
6. 
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DETAILED DESCRIPTION OF PREFERRED EMBOD- 
IMENTS 

As shown in Figure 1 , an image-based touch sys- 
tem 100 includes a display terminal 110 having a dis- 
play screen 112, an image processing system 120, and 
a storage subsystem 130. A camera 140 is centrally 
mounted above the display terminal 110 such that the 
lens 141 is substantially pointing downward. The optical 
axis 142 of the lens 141 is substantially perpendicular to 
the plane of display screen 112, and slightly in front of 
the screen 112. 

The image processing subsystem 120 includes a 
video board 122 connected to the camera 140. The 
video board 122 includes an analog-to-digital (A/D) con- 
verter, and a frame buffer (FB) for storing frames. The 
image processing subsystem 120 also includes an 
image processor 126 and an image buffer (IB) 125. The 
image processing subsystem 126 can include one or 
more central processing units (CPUs) connected to 
local dynamic random access memories (DRAM) by 
busses. The storage subsystem 130 can be used to 
persistently store programs and data used by the image 
processing subsystem 120 in a database (DB). 

The operation of the system 100 is now described 
also with reference to Figures 2 and 3. The image 
processing subsystem 120 creates an image 200 in the 
image buffer 125 for display on the screen 112 of the 
terminal 110. The image 200 includes a plurality of vis- 
ual option "buttons" 201-205 with appropriate legends, 
and an optional an instruction field 210. 

As the image 200 is displayed, the camera 140 
periodically captures or "grabs" frames, e.g., frame 300 
of Figure 3, representing the scene, generally indicated 
by reference numeral 10, in front of the display screen 
112. The scene 10 is converted to digital signals which 
are stored as frames in the buffer 121 as a regularized 
pattern of pixels 301 , for example, an array of 640 x 480 
pixels. Each pixel 301 has an intensity value corre- 
sponding to a level of optical illumination of the scene. 

Figure 3 shows a "reference" frame 300 which is 
captured by the camera 110. The reference frame 300 
includes three general portions 310, 320, and 330. The 
portions 310 (hashed) corresponds to the view of the 
top of the display terminal 1 1 0. Portion 320 corresponds 
to a narrow strip of the scene 10 immediately in front of 
the screen 1 12, e.g., the portion of the scene 10 which 
is closest to the optical axis 1 1 2 of the camera 140. The 
interface between portions 31 0 and 320 generally corre- 
sponds to a plane just in front of the surface of the 
screen 1 12. As will be described in further detail below, 
the width of the portion 320 can be adjusted to match an 
appropriate level of sensitivity to an actual "touching" of 
the screen 1 12. The portion 330 is that part of the scene 
10 the farthest away from the screen 112. The pixels 
301 of the reference frame 300 can be stored by the 
processing system 120 for later use. 

The portion 320 of the frame 300 can be subdivided 


into a plurality of touch zones (TZ) 321-325 which are 
substantially aligned in front of the displayed visual 
option buttons 201-205 to provide "activation" areas. 
Each of the 2ones 321-325, for a particular visual button 
5 size, can be about 64 x 64 pixels. 

In response to viewing the image 200 on the display 
screen 1 12, a user of the system 100, "points" at one of 
the visual buttons 201-205 using a finger or a pointing 
device. As the user is pointing at the screen 112, the 
w scene 10 is captured as a sequence of frames by the 
camera 1 14. As described below, the rate of frame grab- 
bing can depend on how fast the scene 10 is changing. 

The selection frames are compared with the refer- 
ence frame 300 to determine differences in pixel inten- 
ts sity values in the touch zones. Pixel differences can be 
determined using well known image registration tech- 
niques as described in U.S. Patent No. 4,644,582, 
Image Registration Method, issued to Morishita et al., 
on Feb. 17, 1987, 5,048,103, Method for Automatic 
20 Resetting of Images, issued to LeClerc et al., on Sep. 
10, 1991, 5,067,015, Method of Processing Video 
Imag e D ata for use in , the Stor age ^ . Transmiss ion of 
Moving Digital Images, or as described in Re gistration 
of Images with Geometric Distortions, by A. Goshta- 
25 bashy, IEEE Transactions on Geoscience and Remote 
Sensing, Vol. 26, No. 1 , Jan. 1988. Pixel differences can 
be used to detect localized "motion" or optical flow in the 
scene 10. 

The rate at which frames can be grabbed can be 
30 determined by the relative total pixel difference between 
any two successive frames. If the pixel difference 
becomes larger than some predetermined threshold, a 
new reference frame can be grabbed for comparison 
with subsequent selection frames. The process of grab- 
35 bing a new reference frame can be considered a recali- 
bration step. 

For example, selection frame 400 of Figure 4 shows 
a hand and finger 410 entering the portion 330 of the 
scene 10 well away from the screen 112. Any substan- 

40 tial pixel motion in this portion exceeding a predeter- 
mined threshold can trigger a closer examination of the 
selection frames. 

Selection frame 500 of Figure 5 shows the finger 
410 entering the touch zone 332. Pixel motion in this 

45 particular zone corresponds to the user pointing at the 
visual button 202 on the screen. Actual activation of a 
predetermined application processes of the system 100 
can be triggered by the pixel motion in, for example, 
zone 322 "reaching" the interface 501 between zone 

so 322 and portion 310. Sensitivity to actual touching of the 
screen 1 12 can be adjusted by varying the width of the 
portion 320. For example, if the touch zones 321-325 
have a pixel size of 64 x 8 application actuation can be 
made to approximately coincide with the actual instan- 

55 taneous touching of the screen 1 12. 

As an advantage, the image-based touchscreen as 
described above can be adapted to almost any display 
terminal without a physical modification of the terminal 
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itself. The camera 140 can be mounted on a bracket 
attached to the terminal, or on a tripod or other similar 
camera mounting support mechanism separate from 
the terminal. 

This type of arrangement is well suited for a public 
kiosk which may already include an imaging subsystem 
for sensing the presence of a person in the kiosk or 
information booth. 

It should be understood, that the camera 140 can 
be mounted on the side of the terminal 1 1 0 with the opti- 
cal axis 142 having a horizontal orientation. In this case, 
the visual option buttons can be arranged in a vertical 
pattern. In an alternative embodiment, as shown in 
schematically in Figure 6, a mirror 610 is also mounted 
on the side of the display terminal 140 so that the mirror 
is in the scene 1 0 viewed by the camera. The mirror 1 40 
is angled, for example at 45° with respect to the optical 
axis, such that a first angular portion 620 of the scene, 
e.g., the portion 620 between solid ray lines 621-622 
lines captures a horizontal view of the scene 10, and a 
second angular portion 630 between dashed ray lines 
631-632 captures the vertical view of the scene as 
shown in Figure 1 . In this case, the image displayed on 
the terminal 120 can be a two-dimensional array of vis- 
ual option buttons 601-609. 

Figure 7 shows the corresponding reference frame 
700 In the frame 700, a portion 720 of the scene is 
viewed horizontally across by the camera 140 via the 
mirror 610, and the portion 730 is directly viewed verti- 
cally down from the camera. The portion 720 can 
include touch zones 721-723, and the portion 730 
includes touch zones 731-733. A modified image regis- 
tration technique can be used to simultaneously detect 
pixel motion in zones 721-723 and zones 731-733. Pixel 
motion in zones 721 and 731 corresponds to button 601 
being touched, while pixel motion in zones 723 and 733 
corresponds to touching button 609, and so forth. 

It should be understood that the mirror 610 can be 
mounted at other angles which are sufficient to detect 
localized motion for pixel portions in a two-dimensional 
arrangement using triangulation techniques. Two- 
dimensional motion detection can also be done using 
two cameras mounted at different angles with respect to 
each other. In this case, each camera can capture a dif- 
ferent view of the scene, and the frames of the two cam- 
eras can be calibrated and compared with each other to 
determine two-dimensional pointing or touching activi- 
ties 

It is understood that the above-described embodi- 
ments are simply illustrative of the principles of the 
invention. Various other modifications and changes may 
be rrade by those skilled in the art which will embody 
the principles of the invention and fall within the scope 
thereof. 

Claims 

1 . A touch activated computer system, comprising; 


a display terminal including a screen for dis- 
playing a computer generated image; 
a camera oriented and mounted for capturing 
frames representing a scene in front of the 
5 screen, each frame including a plurality of pixel 

values; 

means for comparing the pixel values of a ref- 
erence frame with the pixel values of a selec- 
tion frame, said camera having an output 
10 connected to the comparing means; and 

means for selecting a computer application to 
execute, if the difference between the pixel val- 
ues of a predetermined portion of the reference 
frame and a corresponding portion of the 
15 selection frame exceeds a predetermined 

threshold. 

2. The system as in claim 1 further comprising: 

20 a frame buffer for storing the pixel values of tie 

reference and selection frames. 


3. The system of claim 1 wherein the means for com- 
paring includes a processor for registering images 

25 represented by the frames to determine the differ- 
ence of the pixel values. 

4. The system of claim 1 further comprising: 

30 means for partitioning the reference and selec- 

tion frames into a plurality of touch zones, and 
means for comparing the pixel values of a par- 
ticular touch zone of the reference frame with 
the pixel values of a corresponding touch zone 

35 of the selection frame to select a particular 

computer application to execute. 

5. The system of claim 1 further comprising: 

40 means for mounting the camera so that the 

optical axis of a lens of the camera is substan- 
tially aligned with a plane of the screen of tie 
display terminal. 

45 6. The system of claim 1 wherein the computer gener- 
ated image includes a plurality of visual option but- 
tons, each visual option button corresponding to a 
predetermined computer application. 

so 7. The system of claim 5 further comprising: 

a mirror mounted in a plane passing through 
the optical axis of the lens, the mirror to reflect 
a view of the scene to the camera. 


55 


8. The system of claim 7 wherein the plurality of visual 
option buttons are arranged in a two-dimensional 
pattern. 
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A computer implemented method for activating a 
computer application including the steps of: 

displaying an image on a screen of a display 
terminal; 5 
capturing frames representing a scene in front 
of the screen, each frame including a plurality 
of pixel values; 

comparing the pixel values of a reference 
frame with the pixel values of a selection frame; 10 

and 

selecting a computer application to execute, if 
the difference between the pixel values of a 
p rec ietermined portion of the reference frame 
and the pixel values a corresponding portion of is 
the selection frame exceeds a predetermined 
threshold value. 
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